Tombstones for deleted contracts - #155
Conversation
…hema and indexer utilities
|
@githoboman Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
Miracle656
left a comment
Miracle656
left a comment
There was a problem hiding this comment.
The tombstone model is nicely designed — contractId-unique so first-detection-wins and re-detection is a no-op, createMany with skipDuplicates for idempotency, the "unknown liveUntilLedger is never treated as expired" guard, and thorough unit tests on tombstoneFor / detectExpiredContracts. But there's a gap that blocks merge:
The feature is never invoked — it's dead code. detectExpiredContracts / tombstoneExpiredContracts are only referenced by their own tests; this PR doesn't modify src/indexer.ts (or any poll/cron path), so nothing ever calls them against live ledgers. As shipped, the migration adds a ContractTombstone table that stays permanently empty and downstream consumers watching it never get a signal.
To land this, please wire the detection into the indexing loop:
- Call
tombstoneExpiredContracts(contracts, currentLedger)frompollOnce(or a periodic job), sourcing each tracked contract's instanceliveUntilLedger. If that requires an RPC/getLedgerEntries call that isn't available in the poll path yet, that's the missing piece — either add it or split it into a follow-up and note the intended trigger here. - A small test asserting a row actually lands in the table when an expired contract flows through the wired path.
Also please rebase onto latest main — #154 landed with the lockfile (@emnapi/core) reconciled, so rebasing will clear the npm ci desync this branch inherits. Solid foundation; it just needs to actually run.
…etwork The detection logic was complete and well tested but nothing ever called it: `detectExpiredContracts` / `tombstoneExpiredContracts` were referenced only by their own tests, so the migration added a `ContractTombstone` table that would have stayed permanently empty. That failure is invisible — a detector that never runs looks exactly like a chain on which nothing has expired. Wiring: - `maybeTombstoneExpiredContracts(loop, currentLedger)` runs inside the poll loop on its own cadence, `TOMBSTONE_CHECK_EVERY_CYCLES` (default 100 ≈ 10 min at a 6s poll). Each check costs one RPC call per unique watched contract, and a contract TTL is measured in weeks, so checking every cycle would multiply the RPC budget by the size of the watch list for no benefit. - Its own counter, separate from `pollCycleCount`. Sharing one would make whichever cadence is shorter starve the other, since the prune resets it. - Failures are caught, not propagated: a missed liveness check is retried next cadence, whereas a loop that dies on an RPC hiccup stops indexing entirely. - Extracted as an exported function rather than left inline, so the wiring is assertable. Six tests cover it, including that the check recurs rather than firing once and that a failed check does not stall the loop. Network scoping, which this branch predates: - `ContractTombstone` gains a `network` column, `@@unique([network, contractId])` instead of a global unique on `contractId`. The same contract id exists on both chains with different TTLs, so a global unique would let a testnet expiry permanently suppress the mainnet tombstone. - The migration is renumbered to 20260901120000, after 20260829120000_add_network. That migration back-filled `network` onto the tables that existed when it was written; this one did not, so ordering it earlier would have left ContractTombstone as the only network-blind table in the schema. - `fetchLiveUntilLedger` now calls `getRpc(network)`. Reading a mainnet contract's TTL off the testnet RPC fails the lookup, returns null, and the "unknown TTL is never expired" guard then means the mainnet loop silently never tombstones anything. A test pins that the fetcher is asked for the same network the row is tagged with. tsc clean; full suite 351 passed.
Miracle656
left a comment
Miracle656
left a comment
There was a problem hiding this comment.
Approved and merging — I did the wiring myself (5505a2b) and rebased onto main, since the wave has closed.
Everything I praised in the first review still stands, and the design held up under the change: the pure isExpired / tombstoneFor split, the injectable TtlFetcher, createMany with skipDuplicates for idempotency, and above all "a null TTL is never treated as expired" — refusing to tombstone on missing information is the right call, because the alternative is emitting a "contract gone" signal every time an RPC call fails.
The blocker was that nothing ever called it, and that is a failure mode worth naming: a detector that never runs is indistinguishable from a chain on which nothing has expired. The table stays empty, no test goes red, no error is logged. It would have looked like it worked indefinitely.
Wiring:
maybeTombstoneExpiredContracts(loop, currentLedger)now runs in the poll loop on its own cadence —TOMBSTONE_CHECK_EVERY_CYCLES, default 100 (~10 min at a 6s poll). Each check costs one RPC call per unique watched contract, and a contract TTL is measured in weeks, so checking every cycle would multiply the indexer's RPC budget by the size of the watch list to notice an expiry ten minutes sooner.- It gets its own counter, not
pollCycleCount. Sharing one would make whichever cadence is shorter starve the other, since the prune resets it — the longer job would never reach its threshold. - Failures are caught rather than propagated. A missed liveness check is retried next cadence; a loop that dies on an RPC hiccup stops indexing entirely and does not recover.
- I extracted it as an exported function instead of leaving it inline, specifically so the wiring is assertable. Six tests, including that the counter resets so the check recurs rather than firing once forever, and that a failed check does not stall the loop.
Network scoping, which this branch predates:
Your branch was cut before #159 added the network dimension, so a few things had to change:
ContractTombstonegains anetworkcolumn and@@unique([network, contractId])in place of the global unique oncontractId. The same contract id exists on both chains with different TTLs — a global unique would let a testnet expiry permanently suppress the mainnet tombstone, andskipDuplicateswould swallow it silently.- The migration is renumbered to
20260901120000, after20260829120000_add_network. That migration back-fillednetworkonto the tables that existed when it was written; this table did not exist yet, so leaving it dated June would have madeContractTombstonethe one network-blind table in the schema. fetchLiveUntilLedgernow callsgetRpc(network)rather thangetRpc(). This one is nasty in combination with your null guard: reading a mainnet contract's TTL off the testnet RPC fails the lookup, returnsnull, and "unknown TTL is never expired" then means the mainnet loop silently never tombstones anything. The guard is still right; it just needs the fetcher pointed at the correct chain. There's a test pinning that the fetcher is asked for the same network the row gets tagged with.
.env.example documents the cadence, including that 0 disables the check.
Verified: tsc --noEmit clean, full suite 351 passed.
Good foundation — it just needed to actually run.
…s by network
Closes the over-capture: `mint` and `burn` were accepted as LP-share events
from every contract. Those are generic SEP-41 events that every token emits,
so every USDC mint and every ordinary token burn was being written into
LpShareTransfer with poolId = that token's contract, double-stored beside its
own token-transfer row. `deposit`/`withdraw` are pool-specific and stay
unconditional — emitting them is itself a claim to be a pool.
Bare mint/burn is now only honoured from a contract already established as a
pool. A contract joins that set by:
- emitting an explicit deposit/withdraw, in this batch or a previous one
(parseLpShareEvents scans the batch for the explicit dialect before
decoding anything, so a pool's first deposit and the mint alongside it are
not split by their order within one batch);
- already appearing as a poolId in the table, loaded at loop start so a
restart does not forget and silently stop recording bare events;
- being named in LP_POOL_CONTRACT_IDS, for a pool that only ever emits the
bare dialect.
Tests: a plain token mint from an unknown contract is not recorded, the
explicit dialect is accepted from anyone, a same-batch deposit promotes the
contract, and a carried-over pool keeps working.
Network scoping, which this branch predates:
- LpShareTransfer gains a `network` column with @@unique([network, eventId])
in place of the global unique, and network-leading indexes. A global unique
on eventId would let a testnet event suppress its mainnet namesake, and
createMany's skipDuplicates would swallow it without a trace.
- The migration is renumbered to 20260901130000, after add_network, for the
same reason as the tombstone one: add_network back-filled the tables that
existed when it was written, and this was not one of them.
- upsertLpShareTransfers takes the network; rollbackToLedger deletes LP rows
network-scoped alongside the others.
Merge fixes: dropped the duplicate pre-network ContractTombstone model and the
superseded June tombstone migration (both arrived via the Miracle656#155 stack, which is
now on main), and repaired a stale `totalIndexed` reference that survived the
merge as a bare identifier.
tsc clean; full suite 382 passed.
Miracle656#155 and Miracle656#156 have landed, so every code file this branch carried from that stack now exists on main in a later form. Took main's version of db.ts, indexer.ts, lp-shares.ts, tombstones.ts, schema.prisma and both test files, and dropped the two superseded June migrations. What remains is docs/event-reference.md plus the .gitignore change that makes it visible: docs/ was ignored wholesale, which is why every doc in this repo had to be force-added. Replaced that with a targeted ignore for docs/openapi.json, which is generated by npm run docs:openapi alongside the tracked root copy. Verified the doc against the code rather than reading it: KNOWN_EVENT_TYPES matches decoder.ts:7 exactly, and every base64 ScVal in the worked examples decodes to the value the doc claims — topic0 'transfer', the two G-addresses, and 1000000000 stroops. tsc clean; full suite 382 passed.
2 participants
mplemented contract liveness tracking with tombstone emission on storage TTL expiry, matching the codebase's existing conventions (modeled closely on sac-detect.ts).
Closes #143
Files changed
New: src/indexer/tombstones.ts — the core module:
isExpired(liveUntilLedger, currentLedger) — pure expiry rule. A contract is live through its liveUntilLedger, so it's only expired once currentLedger > liveUntilLedger. A null/unknown TTL is never tombstoned.
tombstoneFor(liveness, currentLedger) — pure: builds a tombstone record or returns null.
fetchLiveUntilLedger — reads the contract instance entry's liveUntilLedgerSeq via getRpc().getContractData(...), exactly like SAC detection reads the executable. Wrapped in an injectable TtlFetcher type for testing.
fetchLiveness / detectExpiredContracts — de-duplicate contract IDs, one RPC call per unique contract.
insertTombstones — idempotent createMany({ skipDuplicates: true }) keyed by the unique contractId (first expiry detection wins).
tombstoneExpiredContracts — the end-to-end entry point: detect + persist.
New: src/tests/tombstones.test.ts — 15 tests over a TTL fixture (live / expired / border / unknown contracts) with an injected fetcher and a mocked Prisma client. No network, no DB.
Schema + migration — added the ContractTombstone model to schema.prisma (unique contractId, liveUntilLedger, detectedLedger, indexed by detectedLedger) plus the matching SQL migration.
Acceptance criteria
✅ Tombstones inserted on expiry detection — tombstoneExpiredContracts / insertTombstones.
✅ Tested with fixture — fixture-driven suite, 15/15 passing.
Verification
npx jest src/tests/tombstones.test.ts → 15 passed
npx tsc --noEmit (and the test tsconfig) → clean, exit 0
npx prisma generate → client regenerated for the new model